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ABSTRACT 



The first section of this paper considers what is 



understood by evaluation in the field of project design for English 
language teaching (ELT) • Published accounts of ELT projects are 
examined for their policy on evaluation and a list of dominant themes 
in project evaluation is drawn up. The second section reviews 
theoretical issues involved in project evaluation and seeks to 
summarize what might be included in a system of evaluation conceived 
as being central to the design and implementation of a project. The 
third section discusses the idea that project designers need to take 
account of the impact of the innovations they introduce. 
Consideration is given to how this aspect of project design might 
influence the forms of evaluation employed by project designers. The 
fourth section looks at a particular ELT project from a teacher's 
point of view, considering the forms of evaluation employed in the 
project and the extent to which they match the features outlined at 
the end of the second section. The fifth section offers suggestions 
for planning the evaluation, and particularly the on-going 
evaluation, of projects in ELT design. (JL) 
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0 INTRODUCTION 

Evaluation is a much debated term. Different people use it to mean 
different things, and different aspects of it are considered important by 
different pt\>ple. Tlie first section of this pa perconsiders what is understood 
by evaluation in tlic field of project design for English language teaching 
(ELT). Published amjunts of ELT projects are examined for their policy on 
evaluation and a list of dominant themes in project evaluation is drawn up. 
In tlie light of these themes, the second section goes on to review theoretical 
Issues involved in project evaluation and seeks to summarize what might 
be included in a system of evaluation conceived as being central to the 
design and implementation of a project. It considers in particular the 
advantages of on-going evaluation. Tie third section of the paper discusses 
the idea that project designers need to take ajcount of the impact of the 
innovations they introt^'ce. Consideration is given to how this aspect of 
project design might i nfluonce the forms of evaluation employed by project 
designers. The fourth section looks at a particular ELT project from a 
teacher's point of view, considering the forms of evaluation employed in the 
project and the extent to which they match the featttres outlined at the end 
of the second section. Rnally, drawing on the conclusions of the previous 
four sectichis, the fifth stvtion offers some suggestions for planning the 
evaluation, atiJ particularly the on-going evaluation, of projects in ELT 
design. 



An earlier version of this paper wat submitted in fulfilment of the 
requirements tif the M.Phil. in Applied Linguistics, Trinity College, 
Dublin, September 1989. 
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1 EVALUATION IN PROJECT DESIGN: A REVIEW OF PROJECT 
DESCRIPTIONS 

This section examines the published descriptions of eight projects in 
order to obtain a general idea of what aspects of evaluation are raijicd by 
different writers working on different types of projects. The projects were 
carried out in a number of different countries and ranged frtim primary to 
postgraduate level, from institutional to narional scale, from the wriHngof 
a course for a particular group of learners to course book writing, teacher 
training and research. The published descriptions all make some reference 
to project evaluation and attempt to incorporate it in a project plan in some 
form or other. We are particularly interested to discover 

- what importance is currently accorded to project evaluation; 

- what aspects of project evaluation are considered important; 
v/hal form the evaluation takes; 

- how it is implemented. 

1.1 Study Skills in English (Candlin, Kirkwood and Moori? 1977) 
This article reports on a three-week intensive course in English for 
students coming to Britain in order to take postgraduate courses. The aim 
was to design pre-sessional courses for the areas of Engineering, Econom- 
ics, Urban Planning, and Foreign Service. The article describes the issues 

thataroseintheplanningand implementation of thecourse. Evaluation was 
built into the model of course design at the final stages in a planning 

procedurewhjchconsistedofsixstages:objectives;syllabusconlcnt;course 
programme; teaching; evaluation of shjdents; cvaJuation of course. 

In the model, evaluation of students feeds back into the teaching 
component, while evaluation of the course feeds back into the objectives 
component. However, we are given no real indication as to how evaluation 
is to take place or who should be responsible for it, nor is there any 
description of a systematic evaluation procedure. 

1.2 Education, ideology and materials design: a Tpnzanian experi- 
ence (Brumf it 1980) 

This report describes a project in syllabus design, and ultimately mate- 
rials design, in Tanzania, Brumfit emphasizes that a main feature of the 
project was the exchange of ideas between the curriculum development 
unit, the inspectorate, teachers in schools, and teacher trainers, with a view 
to obtaining regular feedback about tht course and its progrt^js in order to 
insHtute change. The idea was that this feedback and the subsequen t change 
should be "principled rather than random" (Brumfit 1980, p.l66). 

The machinery provided to support this evaluation prmtxiure was a 



regular pmgramme of in-service courses for teadiers, the appi^ntment of 
"ccKJrdinalors*' (teachers who taught and acted as advisers) in the various 
zom^of thecountryjocal iixlependent English language teachers' associa- 
tions, and a national organisation of English teachers which produced a 
bulletin. These bodicsallowed for the discussion, criticism, and Jissemina- 
tion of ideas concerning the project and its evaluation. 

A syllabus together with an outline of the principles behind it was 
circulated to all participants and acted as a working document. Teachers 
could follow it, or disagree with it and mak^- their own changes, or take its 
ideas and develop them further In sum, it provided a discussion document 
for all concerned. The intention of the project di^igners was that the 
document would be revised and improved as a result of the feedback 
coming through the channels specified above. Thus the document was seen 
as providing a framework for change and development rather than a fixed 
scheme. 

IJ Planning a project the KELT project. Sierra Leone (Hayes 1983) 
This project aimed to support and co-ordinate the production of a 
syllabus for English at primary level, materials to match, an entrance exam 
to reflect the syllabus, a teacher training syllabus based on the project, and 
an in-service scheme. The planning was based on three principles, the first 
of which was flexibility: ''We tried not to strangle ourselves with our own 
project specifications, in the knowledge of what can so easily happen to the 
best-laid plans. Planning is an on-going process in the life of any project, in 
that it must respond to the inevi table changes in the situation. Our priority 
is a strong framework which can cope with any necessary changes of 
activity as the years go on, which can respond to any new need, which can 
survive when the expatriates move out" (Hayes 1983, p25). We conclude 
from this that changes to the original project plan are desirable and that the 
framework should be designed to cope with change. In other words, 
evaluation should take place throughout the project and feed back into its 
structure. But how exactly this should happen we do not leam from the 
report. 

1.4 The English Language Centre, Jeddah (Roe 1980) 
This report describes the work of the English Language Centre, which 
provides English language courses for students of the colleges of mixlicine 
and engineering at the King Abdul Aziz University in Jeddah. Roe suggests 
that various methods of project evaluation can be used: 

- a)mparison of the succt*ss of one programme with another in terms 
of students' rc*sults (expressc^d in bands); 

- comparison between classes within courses; 
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- comparison between the prepress of a group in one semester and its 
progress In previous senators. 

Qearly Roe is concerned diiefly with evaluation of student perform- 
ance, and seems to |udge the success of a couree on this alone. 

1.5 TTie Francophone Primaiy Project Cameroon (Wilson and Harri- 
son 1983) 

This project aimed to introduce English into the last three classes in all 
primary schools in Camerocm. It involved the training of appro)(tmately 
6,000 teadiers through a one-year pre-service course and in-service cou nses. 
A major part of die project was the production of three textbcwks. 

The original project started in 1975, and according to Wjls<m and 
Harrison was not a success. They blame its failure largely on: 

- poor initial project design; 

- lack of consultation during initial planning; 

- lack of observation of classroom procedures in the early stages of the 
project; 

- lack of rigorous trialling of materials; 

- the fact that the writers had no experience of teaching the materials; 

- no dose examination of teacher supply figua^. 

On the basis of this experience, Wilson and Harrison propose a model 
consisting of a series of steps necessary for successful project planning. 
Evaluation plays an important part in this at two stager 
(i) Pre-trialling of materials, where the writers themselves try out 
sample materials. The aim is to assess the suitability of particular 
activities- to calculate what might be covered in one lesson; to judge 
if student teachers could handle materials; and to weed out ptilitical 
a»^d cultural errors, 
(il) Formal trialling of materials. Wilson and Harrison stress the impor- 
tance of including all categories of teachers— traincHi and untrained, 
experienced and inexperienced, gcxxl and mcdk crc, thoM^ with and 
those without support. 
Wilson and Harrison suggesr that il is necessary to have a checklist of 
information required on theappropriacy and effectiveness of the materials. 
The methods they suggest for data collection are questionnaires after every 
five lessons, discussions with the teachers, and observation of teachers 
using the materials. A final aspeci of evaluation that they mention is a pf^st- 
course assessment of the materials. They feel that this should involve 
evaluation of pupil pcrforniance through criterion-referenced tests. 



1.6 The English Language Textbook Project - Somalia (Clarke, 
Hawkes, Pritchard, and Smith 1983) 

This project concerned the design of a textbook for secondary level 
English teaching and the writing of a scientific English course, plus a reader, 
for students undertaking vocational and technical training through the 
medium of English. The project also involved tlw setting up of pre-servicc 
and in-service training courses for teachers who were to use the textbooks. 

The first attempt at evaluation was made during the writing of the 
materials. It was carried out by trainee teachers, wto as part of their degree 
had to research and write papers on the textbook pro)ect. They looked at 

- potential problems for Somali teachers and students with the new 
materials; 

- the rationak' Mtind the textKxik; 

- the economics of textbtx)k production. 

Clarke etal. make the point thatsystematicandon*going evaluation was 
desirable since this project represented a new direction in English language 
teaching in Si)malia, Furthermore the projected lifetime of the textbouk was 
1U-15years. "Evalu«itiim, then,isconsidered by the team to be a core activity 
inali stages of theproject. Evaluated data provides or has provided the basis 
fi>r judgements to be made on: 

(a) the nature of product; 

(b) the product quality and quantity; 

(c) the operational success of the product; 

(d) training requirements for serving teachers and teacher- trainees; 

(e) the impact of the programme on English language teaching and 
learning in Somalia in the light of the project objectives'' (Clarke etal. 
1983, p.65) 

The instruments of evaluation used in trialling were as follows: 

- Reports of classn>om observation on the ability of teachers to use the 
materials sucwssfuUy. This should indicate areas in which pre- 
servia^ training is necessary and assess student response to materi- 
als. 

- UscTs' rept)rts filkni in by each teacher during trialling to provide 
data on pace and specific problem areas encountered. This should 
provide a tesson-by-lesson evaluation which indicat«?s teachers' 
attitudes U) new materials. 

' A questionnaire filled in by teachers on aimpletion of the trialling 
package to provide data on student motivation, teacher attitudes to 
teacher notes and materials, and specific problem areas. 

- A questionnaire filled in by students to gauge their reaction to the 
new coursi>s. 

- Riy,uhir infonnal interviews with teachers basi^J on their qui^stion- 
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naires and reports to elaborate on the data. 
-Regular samples of students' work to evaluate their success in 
following the course. 

- Tests based on the perfonnance objectives of she textbook to assess 
the pedagogic success of the materials. 

The resulting data was used: 

- to improve the production of subsc»]uent biHiks; 

- to improve the provision of pre/in-service teacher training; 

♦ to improve support for the teacher through belter teacher ranes and 
guides; 

- to assess the overall success of the project so as to inform projects of 
a similar nature elsewhere. 

The constraints on this evaluation were as follows: 

- There was a pressing need for materials, and this meant that the first 
book was not trialled prior to its publication. Therefore, information 
obtained from subsequent trialiing was not used to change the 
materials but was fed into the teacher-training programme to help 
teachers cope with tfie flaws, 

-Ccf^aphic distance and pot>r communicititins meant thai it was 
difficult to monitor the implementation of the programme. 

- Factors such as lack of adequate facilities (e.g., equipment and 

rooms)andinadcqu3te training of teachersinlhcpast had l»>be taken 
into account. 

1.7 The Ain Shams University Project, Egypt (Bowers 1983) 
The primary aim of this project was to pmvidc undergraduate curricu- 
lum revisi(>n in TEFL, postgraduate TEFL training, test development, 
conferenceand seniin3rorganization,andresearchactivities. Bowers kx>ks 
at how the mismatch between project planning and project performance can 
be minimized. The main thrust of his argument is that any change in a 
system will initiate changes In all other sectors of that system. This is what 
he refers to as the "spider's web" effect (Bowers 1983, p.l(X)): when you 
touch one part, the whole vibrates. Therefore- an allowance for change n .^eds 
to be built into a project at the planning stage, and every project shouk' be 
aware of its effect on other sectors, 

Bowers(l983,p.ll5)suggcst5that the following opcrati«inalprtx:edures 
should be included in the planning of a project in order to cater for the 
spider's web effect: 
(i> A problem or need is recognized, 
(ii) The real problem or need is identified. 
'm) Possible solutions are considered, 
(iv) The implications of each are followed through. 

■> 



( v) A course of action is decided upon and this becomes the operational 
design. 

(vi) Personnel and resources are negotiated, 
(vit) AH intea'sted parties are involved, 
(viii) Implenientation comnumccs. 

(ix) Formative evaluation and regular reporting takes place. 

(x) At an appropriate jwint a summative evaluation is carried out and 
the cycle is repcated/adjusted/terminated. 

(xj) Experience is disseminated. 

Bowers stresses the need fore jmmunication between the project team, 
the administn tfve and financing authorities, teachers, aiui trainers. In 
addition, the project team needs to communicate within itself, to ackfK)wi- 
edgeand incorporate different views. Therefore theremustbe ''clarification 
of the criteria and processes for on-going evaluation" (Bowers 1983, p.l 16), 
and this needs to be done at the outset, and to be specific. At the same time 
it should be flexible to a How for adjustments resulting from policy changes 
or practical constraints. Bowers says that there Is little to be gained in saving 
evaluation for the end of the project. Evaluation at this stage might show 
weaknesses, but it would not be possible to correct them; it might show 
strengths, but it would be too late to develop them. 

On the question of who should be responsible for evaluation. Bowers 
(1983, p.l 17) points out the advantage of the implementer, e.g. teacher, 
playing the role of evaluator. He stresses the useful insights that an 
"insider", such as a teacher, would have, which the outsider, e.g. foreign 
expert, might not have. He further stresses the need to exploit the contribu- 
tion that different parties can make to the curriculum prcxx?ss. Finally, he 
suggests (1983, p.l 17) that it is normal and desirable that project expecta- 
tions and project r^^ults should differ, since curriculum development is a 
dynamic process, created by the interaction between innovation and the 
context. 

1.8 The University of Nairobi Project (Brown and Hirst 1983) 
IWs project was si*t up in the Language and Study Skills Unit of the 
University of Nairobi to pre vide an improved study skills course for 
students at the university. Tlie main types of evaluation undertaken by the 
unit are: 

(i) pre- and post-testing of courses written by the members of the unit 

in terms of the objectivc»s of each course; 
(it) evaluation of the work of the unit in terms of its effect on student 
performance in their own subject areas. 
AiYording to Brown ami f lirst, evaluation of a ci>ur.s4» rcquiri^sa pre test 
of two groups of students, those taking the course and a contn)l group of 
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students not taking It. These groups should be matched in terms of linguistic 
skill, the subject courses they are taking, and Initial subject knowledge. Ihe 
post-test should bea subject test, devised by the subject lecturer. Brown and 
Hirst point out some constraints on thb approach. For example, there are 
practical problems in ptting a suitable control group and in co-ordinating 
the post-test with subject teachers. It is also difficult to determine whether 
improvement is due to the shidy skiilscourse or simply to attendance at the 
university (Brown and Hirst 1983, p.l43). 

B Town and Hirst suggest that it is necessary to test not only whether 
students have leamt anything in study skills classes, but also If what they 
have leamt is useful and if they can transfer i\ successfully to their subject 
areas. They say that there are no objective criieria for evaluating this. The 
noethods they suggest are (Brown and Hirst 1983, p. 144): 

(i) judgcmentby membersof thestudyskillsuniton the effectiveness of 
the course; 

(ii) the opinion of students on the ctiurse elicited through a post-course 
questionnaire; 

(ill) the opinions of subject staff elicited through 
•consultation on written work, 

- observation of study skills classes, 

- informal assessment of theeffectsof the course on students' work 
in subject areas. 

Brown and Hirst (1983, p.l46)feehhat evaluation is essential. However, 
they are unhappy about the unclear and unrigorous methods that are 
suggested here. They feel strongly that more research nt\>ds to be done in 
this area to produce more satisfactory methods. 

1.9 Conclusion 

The eight projects presented above were chosen because of the variety 
of settings, aims, and scope they display. They all deal with evaluation in 
some form or other, and they are all relatively recent and tnerefore 
demonstrate up-to^ate practice in this area. Collectively they confirm that 
evaluation is a very imptirtant aspect of project dt»sign and one that all 
project planners have to consider. We iuggest that the foUtn-ing nine 
themes, all of which recur in these descriptions, are centra! questions in this 
debate, are to some degree unres<ilvcd, and arc therefc^re worth further 
Investigation: 
(I) Why include evaluation in project design? 

(ii) What should be evaluated? 

(Hi) What criteria should be used for evaluation? 

(iv) Who should be involved in < valuation? 

(v) What form should evaluation take? 

O 8 
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(vi) Should evaluation be summative, formative, or both? 

(vii) At what stages in the pri^amme should evaluation take place? 

(viii) What channels should be used to feed information back into the 
programme? 

(ix) What kind of amstrainlscan wecxpect loencounlerin the evaluation 
of a programme? 

2 EVALUATION: SOME THEORETICAL CONSIDERATIONS 

Wecontinue oursurvey of theliteratureby turning to three writers. Potts 
(1985), Murphy (1985) and Rea (1983 and 1987), whodeal more specifically 
with theon^tical aspects of project evaluation, Weshail examine their views 
to discover their pi>sitiims vis-^-vis the uinc themw listed at the end of the 
first section; todiscoverwhatkind of arguments ami polemics surrounding 
these issues need to be taken into consideration; and to establish what kind 
of theoretical stance we might adopt on these points. 

2.1 Why include evaluation in project design? 

Potts (1985) divides reasons for communicative curriculum evaluation 
into two types: internal, i.e. those generated by the nature of the aimmuni- 
cative curriculum itself; and external, i.e. those designed to provide evi- 
dence to outside interests about what is happening on the amrse. Potts 
suggests that the first inten.al reason for a fonrwl evaluation component in 
a communicative curriculum is "to sort out priorities, weight them and so 
guide future activity" (1985, pp.24f.). A second reason he gives is to allow 
the growth of a sense of i:chievement in tlie learner. Examples of external 
reasons for evaluation may be course validity, assessment of the progress 
of individual students, placement criteria for future courses, and assess- 
ment of the usefulness of materials. 

The development of the communicative curriculum regards the ques- 
tion of how people learn to use a language as part of the teaching/learning 
process. Both teacher and learner are thus engaged in on-going formative 
research (Potts 1985, p.21). Central to this prcxress are the needs and wants 
of the learner. Pijtts makes the point tliat unlike structural and functional 
curricula^ where the the content of a curriculum is based on an inventory of 
items specified in advance, thecommunicativecurriculumshould allow for 
systematic change to its content. Tf le evaluation procedua* of a communi- 
cative curriculum, therefore, will measure shifts in learners' ncH?ds and 
wants and translate these into learning activity. 

Murphy (1 985, p,4) comments: "Evaluation should bean integral part of 
the working of the curriculum to ensure that what is done is worthwhile, 
necessiiry and sufficient." Like Potts, Murphy gcK»s on to say that the 



curriculum should not be regarded as static, and that evaluation may be 
used as a basis far further curricular change. Thus evaluation "contributes 
toad justing the curriculum as needed, and to ensuring its smooth working" 
(Murphy 1985, p.4). 

Evaluation is one way lo produce information on assessment. This 
involves not on!y tests but assessment of all other elements of the curricu- 
lum. Furtheiin&re, it is a means of producing information for public 
aocountability. This means being able to report accurately on teaching, 
which in turn means reporting on good and bad results. A hirther reason 
for evaluation is to find out more about what actually happens in teaching 
and learning, and thus improve curriculum development. It is therefore a 
means of showing how well or badly a new theory of syllabus design works 
in practice. So what it requires is an empirical approach to new ideas on 
teaching and learning (Murphy 1985, p.9). 

At the moment, the pn^lem is that there is no formal means for teachers 
to find out about the effecHveness of their leaching in an empirical way, 
apart from testing. There are no indications of how teadicrs should 
interpret the results they are getting, and no proposals for altemarive forms 
of evaluation that teachers could use for this purpose. Murphy criticizes 
syllabus designers who show a "give it a whirl and see how it goes " attitude 
to evaluation (1985, p.8). He takes the line that for too long we have been 
happy to rely on reasoned ti.eory for the justification of our syllabuses, and 
for too long have ignored the implicaHons of data from the classroom 
situation. His proposal, therefore, is for a comprehensive, balanced evalu- 
ation, which involves teachers at classroom level. 
Rea (1983, p^9) gives the purposes of evaluation as: 

(i) infon. - tion-gafhering and disseminatit)n; 

(ii) quality control; 

(iii) accountability; 

(iv) progrc^ and achievement. 

Toallow people involved in the project design process to make e.'fccti ve 
decisions, it is necessary to obtain a certain amouni of infomwtion at the 
various stages of the project. So, for example, materials writers will want 
feedback on the success or failure of their work, institutions will want 
information on student achievement, and sponsors will want information 
on the safeguarding of their investment, A further rcasim Rea gives (or 
evaluation is lo monitor what she refers to as the "localisation process" (Rea 
1983, p.93). In projects where there is expatriate involvement. 

According to Rea (1987, p.l50) evaluaHon in project design has essen- 
tially two purpose's. The first purpose is similar to Potts's internal reasons, 
and is intended to provide guidance ft)r subsequent teaching and learning 
by using information derived at different intervals within the teaching and 
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learning process. The second purpc^ is to provide information on the 
adequacy of the curriculum in use to outside interrats, such as clients who 
have commissioned the programme. This dosely parallels Potte's externa! 
reasons for evaluation. Rea ( 1987, pJ49ff*) suggests that a hirther importa^^ 
reason for the inclusion of evaluation in the design of a communicative 
curriculum is the need to validate new, theoretically appealing approaches. 

Thirte seems to be general agreement in the literature that evaluation is 
important for. 

- validation of the curriculum design and of the theoretical concepts 
tx!hind it; 

- inkemal curriculum development; 

- external face validity. 

There is general dissatisfaction with the methodsof evaluation currently 
in use, and a belief that there has been too much stress on evaluation for the 
purpose of assessment of student achievement. In general therefore, the 
whole pur|X)se of evaluation needs to be broadened, and this will have 
implications for all aspei:ts of assessment. 

22 What should be evaluated? 

All three writers under review take the line ^hat what is often assumed 
to be the main concern of evaluation, viz, learner performance, is by no 
means the only thing ihat should be evaluated. Potts (1985, p.22) suggests 
that there must t>e some evaluation of whether what is being learned is 
appropriate to the learners' needs; whether skills are being transferred; and 
if they are not, what rangeof slrategiesare necessary for successful transfer. 
Commenting on the idea of measuring students' progress in termsof a fixed 
target repertoire. Potts says (1985, pJ20) that this "inventory^ approach to 
evaluation is not really possible in a communicative curriculum because of 
the flexible and variable view of language on which it is based. Mastery of 
target language structures and vocabulary is not a sufHdent measure of 
a>!irsesuccessor student performance. Potts(1985, pp^Of.) makes the point 
that the communicative curriculum regards the question of discovering 
how people learn a language as part of the language learning process, which 
means that this too becomes the subject of evaluation to the extent that we 
wish to evaluate the process of language learning. 

Thedemandsof communicative methodology imply that what is learned 
and taught should be "rear' language. Thus for projects in communicative 
language teaching a further subject for evaluation is whether the project 
includes Such language (Potts 1985, p.2I). Another demand made by 
communicative methodc^logy is tliat the real needs of the learner should Ih? 
met. litis rc\]uirisanenipi)asis()n language in use^ rather tlusn on grammar 
or communicative functions. We would therefore want to evaluate the 
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learners' useoflanguage in iheclassroom, ratherlhan mastery ot forms. We 
will also need to consider the extent to which classroom learning is related 
to real communicative needs, as well as the extent to which learners are 
enabled tobringexistingcommunic«iti veccmixrtence to theclassroom. The 
pointwasmadeinthelast section that hamers need tobeawaieoftheirown 
progress. This has implications for what is evaluated and would require 
comparison of learners' previous product with their current product. 

Murphy (1985, p.4) agrees with Potts that it is not just the performance 
of shidenis that we should take as the subject of evaluation, but the various 
factors involved in the teaching and learning process. He details these as: 

- the curriculum itself and the context in which it is set; 

- materials; 

- teaching and learning; 

- attitudes; 

Under teaching and learning we might examine individual learner 
performance through tests, but would also needs look at the different 
techniques and strategiesof learning. Not all of these factors will be of equal 
importance for each type of evaluation, but whatever weighting each factor 
receives, it should be present. Furthermore there is a need to report on 
positive data as well as negative. 

Rea (1983, p.90) also stresses the need to evaluate both the prtxress and 
tiie product of learning. She makes the point that what is evaluated depends 
on what thejnirpose of the evaluation is and for whom and when it is 
conducted. For example, the data might be descriptive and serve an 
explanatory function. It might also be used to serve as a feedback to learners 
and teacher:, and act as a guide to changes in (he future. 

Rea (1987, p.150) talks about using evaluation to validate a curriculum. 
Hare she firstly suggests that materials and tasks should be examined to 
establish whether they meet the original curriculum specifications and the 
needs of the target audience. Secondly she suggests looking at how the 
principles of the curriculum are realized in the achjal learning units. She 
argues that a main function of evaluation is to assess the general suitability 

ofthecurriculumfortheneedsanddemandsforwhichitwasdesigned.Rea 
notes that the emphasis in the evaluation of a communicative curriculum 
has to shift from content materials to learning activities and to the oppt)r- 
tunities presented to learners for the development of their communicative 
abilities. Thus the things that will beevaluated will refer more to the privess 
of learning than to the product. 

On the whole our tiiree authors feci that in the past there has been an 
exclusive concern with the course prcduct. While this may still be of some 
relevance, the most important focus in a communicative curriculum must 
be on the pnxress of learning and teaching. 
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Z3 What criteria should be used for evaluation? 

According to Potts (1985), thereared^fferent types of criteria that can be 
used for evaluation. As v/e have already seen, he makes a broad distinction 
in eval ^ation t>et ween process an i product Product Potts suggests, might 
be further divided into end-product and on-going product. Any of these 
aspects of a project can be examined in terms of four different criteria: 

- initial input; 

- target performance; 

- the learner's previous performance; 

- other learners' performance (Potts 1985, p.34). 

Depending on the context of the project, more or less weighting might 
be given to particular types of criteria. For example, for an English for 
Spivific PurixiH's iVSV) aiurso it is quite likely that tin? end-product, and 
consequently target performance criteria, will be important. Butthe weight- 
ing of the different criteria will also depend on the perspective of the 
evdluator.Forexamp!e,onan English for AcademicPurposes(EAP)course, 
a subject teacher would not have the same perspective on evaluation as an 
EAPteacherand would consequently givea different weighting todifferent 
criteria. In any case, it is important that the criteria are clear to everybody 
involved, including the learner. 

Murphy (1985, p.4) argues that the criteria will depend on whether the 
evaluation is concerned with the product of a course or with its process<». 
If it is with the former, then effectiveness will be the chief criterion, whereas 
if it is with the latter, the chief cri tenon will be efficiency. The ef fe(. a veness 
of product refers to the matching of the actual results of tests with the target 
results. Murphy points out that tWs criterion for evaluation is of limited 
usefulness for the improvement of a course since it d >es not tell usanything 
about the reasons for the discrepancy between expected and actual results. 
Nor does it give us any indication of ways of pin-pointing what it is that 
needs adjusting in order to minimize that discrepancy. Murphy (1985, p.4) 
goes so far as to suggest that this lack of information produces professional 
secrecy and entrenchment among teachers. Where efficiency of process is 
the chief criterion for evaluation, the focus is far less fixed. Murphy (1985, 
p.4) argues that this triterion "is more likely to involve all the gmups 
concerned, to encourage them to cooperate and thus to produce the 
information on which to base development". 

Rea (1 983, p.91 ) stresses t ha t no ma tier what, for whom, or when we wish 
to evaluate, it is the context that is the most important factor in determining 
the criteria for evaluation. Consequently, flexibility and responsiveness 
must be important factors in establishing criteria. As much as possible 
should be predicted about the evaluation procedures of any project, and 
planned into the design dcxument. However, it is not possible to predict all 
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variables. 

Examples of the types of criteria that Rea (1987, p.l64) gives arc: 

- the diagnc^Hc nature of tasks; 

- the coherence of tasks within the social structure of the classroom; 

- the complexity of tasks; 

- how the success of tasks is determined; 

- the transfer of skills from the classroom to the outside world; 

- the extent to which activities in the classroom reflect the real needs 
of the students. 

Criteria for evaluation are generally thought to be closely related to 
context Consequently it is difficult to generalize about criteria in ad vance. 
However, it is considered an advantage to set out in advance as far as 
possible, not only what is going to be evaluated, but also some criteria for 
that evaluation. And it seems very important that ail involved should be 
aware of what these criteria are. 

2.4 Who should be involved in the evaluation? 

In Murphy's view (1985, pp.2f.)oncaim of evaluation is to develop the 
curriculum, so that anyone involved in its optTation can make a valid 
contribution to its evaluation. This includes curriculum planners, teachers, 
learners, and people outside the institution, such as sponsorsand society at 
large. Murphy is particularly concerned to emphasize the potential n>leof 
teachers, who in tlteir function as curriculum managers are concerned with 
the same issues as the curriculum designers. He points out that their 
judgements may be as influential in the process of education as the formal 
findings of the experts. Evaluation, as it exists at the moment, d(X?s not 
exploit this potential. This is because currently available mtxicls of evalu- 
aHon do not provide a suitable framework for the inclusion of teachers. 
Murphy (1985, pp.l5f.) also seen« to suggest that teachers do not have the 
necessary training to assume responsibility for evaluation. 

Potts (1985, p33), on the other hand, emphasizes the fact tAat on-going 
evaluation requires teachers and students together to be engaged in active 
research. He suggests (Potts 1985, p.23) that there is a tension between the 
need to teach languagesauthenHcally and theneed to monitor and evaluate. 
For this reason the burden of evaluation is too great for the teacher to bear 
alone. If evaluation is to work, the learners t(x> must be aware of and 
involved in the evaluation process. Potts (1985, p.38) further stresses the 

importance of suitable matcrialsinthispnxress. He points out the potential 
for confusion over the way in which teachers evaluate learners' perform- 
ance if learners are not aware of the criteria teachers arc using and if they 
remain outside the process. 

Rea (1983, p.90) also stresses the range of people who may be involved 



in evaluatio! j the extent that they may obtain i nfonnation from it on which 
to make decisions— e.g. tutors, materials writen, hc^t institutions, backing 
organisations, or governments. On the other hand/ she argues (19S3, p.90) 
that evaluation can be damaged if the detailed planning stages are handled 
by inexperienced people. The main danger is that they will impose too rigid 
a structure and not allow enough flexibility. However, in terms of who 
provides data for an evaluation, Rea is of the view that tcacl^rs and 
curriculum developers both have roles to play, as well as students and 
sources external to the language programme, for example, in the EAP 
setting, subject specialists (Rea 1987, p.l59). 

The various points of view weha ve been reviewing imply that tradition- 
ally evaluation is the domain of the expert and that therefore there are not 
nnany people who know much about it. It seems to be desirable to widen the 
range of people involved. It also seems important that the frameworks for 
evaluation should be more easily accessible to the non-expert. In sum, there 
is a call for the "demystifying" of evaluation. 

2.5 What form should evaluation take? 

Murphy (1985, p,l5) suggests that tl¥?re are two basic forms that 
evaluation can take: 

(i) evaluation that produces qualiiative results; 

(ii) evaluation that produces quantitative results. 
Whatever method is used, the starting point must be to know what kind 

of tnformatiim is needed and what are the most appropriate waysof getting 
ttiat information. In order to be appropriate, a method must be reliable, i.e. 
it must produce the same result if repeated in similar circumstances; it must 
be val id in the sense tha t i t answers the questions we set out to answer; and 
it must be accessible in the sense that it can be described and replicated. 

These requirements could be difficult to n^t in the classroom and this 
would naturally have implications for Murphy's aim of involving teachers 
in evaluation. Accordingly, a cleariy defined form for the evaluation is 
essential, in combination with procedures for familiarizing teachers with it. 

We have already mentioned a prc^Iem associated with teacher evalu- 
ation of learner performance, namely possible confusion in the way learners 
interpret teachers' evaluation criteria. Potts (1985, p-33) is concerned that 
this pn>blem should be solved and suggests that one way of doing this 
would be to put evaluation in the hands of the learners. This would 
obviously have consequences for the form that evaluation might take. For 
instance, it means that evaluation criteria must be leamer<entred. To 
translate this ob^xrtive into practicable forms and methods, usable by 
learnersand {cMchiTS, could posi'itsown prciblems. Potts (1985, p.33) makes 
the point that materials play a vital role here and can assist students to 
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identity, in an authentic way, what they have done. 
Rea (1983, pm lists possible strategies for evaluation: 

(i) measurement; 

(ii) self-assessn\cnt; 

(iii) d>servation; 

(iv) record/diary keeping. 

In a later paper (Rea 1987, p.l52) she gives us a more elaborate list of 
means that can be used to derive data. She makes two disHnctions, firstly 
between quantitative and qualitative means and secondly, between formal 
and Informal means. Furthermore she emphasizes that a range of different 
procedures, as well as input from a variety of sources, should be used in 
order to obtain reliable and comprehensive information. 

The evaluator will have to make the ultimate decision as to what 
methods will be most suitable, according to the environment, nvhat works 
well in oneenvironment may not produce the desired responses in another, 
different siluaHon" (Rea 1987, p.l52). Examples of the means of collecrtng 
information that Rea suggests are : 

Formal - systematic review of curriculum materials; 

- tests; 

- meeting with instructors; 

- guided questionnaire; 

- structured interviews. 

Informal - generally more open-ended feedback such as reports, un- 
guided and 0|>en questionnaires. 

A finalpoint to make isthatmethodssuitablefor product evaluation will 
tend to Involve measurement, whereas methods that are used for process 
evaluation will more likely be descriptive (Rea 1983, p.97). 

The main conclusion here is that the methcxis ()f evaluation chosen need 
to be appropriate to the situation. It would therefore seem sensible to have 
a bank of mctbods from which to choose, according to the desired criteria, 
according to what is being evaluated. ?nd according to who is evaluating 
and the state of their knowledge. 

2,6 Should evaluation be summative, formative or boUi? 

We have already discussed Rea's (1983) point that different audiences 
need and expect different kinds of information; we have examined the 
process/ product issues that influence what we evaluate; and we have seen 

thatevaluation may bcdescriptiveandservcancxplanatory function or,on 
the other hand, serve as the basis for decision-making. 

Now, another important distinction to make is between evaluation 

which "serves as a feedback to participants in the evaluation process" and 

evaluation which "serves as a guide to future developments" (Rea 1983, 
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p.90). The f dea of guiding future development Uiroui^h evaluation is what 
is commonly referred to as formative assessment^ since this part of evalu- 
ation forms an integral part of the whole idea of pro|ect design develop- 
ment This allows for continuous development atvi change to take place 
within the curriculum. 

Theoth^kindofevaluationprocedt re is summative assessment While 
formative evaluation is concerned witti process, this is concerned with 
product. It is likely to be quantitative in form, and to be used largely for 
decision-making purposes. Examples of this type of evaluaticMi method 
would be formal means of measurement, such as standardized tests and 
examinations. 

A major distinction which Rea (1 983, p.90) makes between the two types 
of evaluation is that formative evaluation is likely to be more subjective in 
nature. She mentions methods of data gathering such as self-appraisal, 
interviews, and (^>servation. This represents a move away from a more 
traditional approach to gathering facts, towardsa new approach which she 
calls information exchange. The implicationsof the two typesof assessment 
are different. Fumiali ve evaluation implies a built-in allowance and desire 
for changp to a programme or curriculum. We *iave already seen that Rea 
recommends flexibility and responsiveness to change and innovation in a 
programme. Formative evaluation provides the machinery for this. 

In her definition of a communicative curriculum, Rea (1987, p.149} 
emphasizes "learner uptake (i.e. behaviour) as something that cannot be 
preordained and predicted". She says that in the past test procedures have 
been too narrowly ''prescribed as quantitative and restricted to the meas- 
urement of the outcomes (i.e. the products) of the learning process''. She 
redefines evaluation in broader terms, one of which is: ''negotiable, arising 
out of the teaching and learning, that can in part be sped ficd in ad vance, but 
will also be oj>enH?ndcd'' (Rea 1987, p.l50). The implication of this is Uiat 
while there is a place for summative evaluation this should not be to the 
exclusion of formative evaluation. We have already discussed different 
forms of evaluation and they can be seen as valid in temis of the different 
functions they serve. For example, there are needs created by the internal 
developmiMit of tlu'curriculum, and neiHJscreatedty public accountability. 
According to Rea (1987, p.l51), both types of ^'valuation can provide 
different kinds of evidence of the effectiveness of programmes. 

Murphy (1985, p.4) similarly tends to equate evaluation of product, 
generally learner performatKc, with summativeevaIuation,and evaluation 
of the learning pro?ess with formative evaluation. His interest in the whole 
question is in the consequences for teaching and curriculum development. 
His view is that formative evaluation, because it is process- rather than 
product-oriented, is more likely to encourage participants, especially teach- 
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ere, toget involved In the process. In this way It is more likely that there will 
be co-operation and openness, rather than professional secrecy. A conse- 
quence of this could be a high level of pjcountability and professional 
awarenessamong teachers (Murphy 1985, pp.14 and 16). This is not todeny 
the usefulness of quantitative and summaHve evaluation, but Murphy 
stresses that up to now an over-emphasis on this aspect of evaluation has 
meant little development in other aspects. And of coui^e he is most 
concerned witft theidea of involvingpeopleother than the outsider /expert 
in the whole p ocess. 

In Potts's opinion, the communicative view of language teaching and 
learning has far-reaching implications for the type of evaluation that needs 
to be adopted. The main implication is that the communicative curriculum 
does not have the easily idenHfiable and quanHfiable internal criteria that 
we find in structural and funcHonal curricula (Potts 1985, p2Q). Similariy, 
while the objecti vecurriculum could be seen as staHc and not vulnerable to 
change as a result of assessment, the communicative curriculum must be 
responsive to evaluation (Potts 1 985, p.22). Potts concludes that an accurate 
procedure for both formative and summaHve assessment is vital. He holds 
that the quality of learning in any course in progress depends on formative 
assessment, while future cunriculum development depends on summative 
assessment (Potts 1985, p.22). 

The three writers under review arc at pains to stress the imptirtance of 
formative assessment, mainly because they feel that it has been neglected 
up to now, because the machinery for it hasnot been a vailable and the need 
for it has not been perceived. At the same Hme there is clearly a need for the 
continuation of summative assessment. 

2.7 At what stages in the programme should evaluation lake place? 

Traditionally, evaluaHon is retrospective and takes place at the end of a 
programme. Thedirection of our argument so far has been that this should 
change. With the stress on formaHve evaluaHon must come change in 
every aspect of evaluaHon. We therefore need to look again at th. ques;:.>n 
of when in a programme evaluation should take place. 

The emphasis on assessment of the process of learning as a basis for 

changewouldsugge.tthatevaluaHonshouIdbchappening throughout the 
programme and no! just at the end. Furthermore, Murphy does not see 
evaluaHon asdistinc. from the language teaching prtxx^ss, but as an integral 
part of it (Murphy 1985, p.l6), ^ 

In this connexion we might consider the model of the curriculum put 
forward by Breen and Candlin (1980, cit. Murphy 1985, p.7), in which 
evaluaHon is a central component. They argue that since evaluaHon is a 
natural part of communicaHon, the evaluation of learners must become a 



continuous process within the curriculum. The point here is that whenever 
there iscommunication in theclassroom there is also evaluation. However, 
the weakness of their lineof argunien t, according to Murphy (1985, p^), lies 
in the lack of systematidty of evaluation provided for in this model If 
learners are to evaluate the grammaticali^, coherence, appropriaten^ 
and intelligibility of utterances in the classroom in a way that provides 
useful feedback to other learners, they will need to be taught how to do it 
Furthermore the question is raised as to whether the kind of evaluation that 
goes on in real communication is different to the kind of evaluation 
necessary for classroom learning. Murphy (1985, p.9) concludes that we 
need to know hiiw such theory works in practice. This means more 
empirical testing of such ideas in the dassroom. 

We return now to Potts's idea of evaluation as t\e "motor of the 
curriculum" (Potts 1985, p,19). He also sees evaluation as a continuous 
process, and the constant driving force behind curriculum development. 
Thus for him too evaluation cannot be relegated to the last stag? of a 
pr(>gramme. Potts advocates giving students some responsibility for their 
own evaluation right from the start. 

Rea (1983, p.88) proposes a three-tier evaluation system— initial, on- 
going, and retrospective. It is not suffident to provide for evaluation wt^^n 
the project has been set ^ip and is running. Instead it should be built in at the 
design stage and begin pricH- to the planning stage. Rea (1987, p,150) makes 
the point that evaluation at different stages in the curriculum process may 
be undertaken for different reasons. For instaiKe, if evaluation is related to 
the process of learning and has a pedagogic function, then it will take place 
during the learning process. On theother hand, if we wish to satisfy external 
requirements and provide the administration with product figures, then 
evalua^on will l>e retrospective and will take place after learning. 

Rea (1 983, p.93) gives an exampleof an evaluation plan where she shows 
that external and in temal requirements need not conflict. To satisfy internal 
pedagogic rcqu irements, regular checksof sti dent performanceareplanned 
into the course in the form of taslcs. At the same time, in order to satisfy 
university requirements, formal tests are planned in also. 

Potts, Murphy and Rea agree that evaluation cannot just be left to theend 
of a course, but must be planned in right from the start. Some assessment 
must be on-goingand occur at regular stages throughout the course so that 
its results can be continuously fed back into the course. 

2«8 What channels should be used to feed information back into the 
programme? 

Potts(1985, p.33) sugginits that ''niuf liple focniback cycles" are necessary 
in order to capitati:ce on the intrinsic communicative role of evaluation 
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within the classroom. The idea is foi feedback from evaluaHon to be 
conducted within an open communicative environment, and, as we have 
already said, this must involve the learners directly, under teacher guid- 
ance. Evaluation, according to Potts, is a "put lie concern*' and should not 

beseenjustin termsof a teacher/learner concem,butasa wholcnegoHaHon 
ivilhin theclassroombetwecn learners, between leamersand materials, and 
soon. 

Accordingly, weconclude thatevaluation feedback is used to inform the 
learning process and to fiirther foster real communication in the classroom. 
Evaluation feedback also has a function as far as moHvation is concerned. 
It is now intended to direct hiture learning activity in so far as it promotes 
on-going debate in the classroom about what is being Icamt and how it is 
being learnt. According to Potts, this in itself may provide a useful and 
authentic source of language learning. 

These ai^guments imply that there need to be many channels within the 
classroom for feedback in all direcHons and involving all elements. It is not 
only a question of the teacher channelling feedback to learners. It is also a 
matter of learners channelling feedback to the teacher, acli viHes and tasks 
acting as a channel to all concerned, learners channelling feedback to each 
other, the learning programme channelling information on the learning 
process to learners, teachcrschanr.eiling feedback to other teachers or other 
people Involved in the programme, and so on. 

We have seen that among Murphy's reasons for evaluaHon is the idea 
that we assess whether current theory works in practice. This means that 
while he assumes that teachers need to lake more responsibility for 
evaluation, the information that Is collected must eventually be channelled 
to those involved in formulaHng the theory on which curriculum develop- 
ment is based. 

For Roa (1983, p 88) proper discussion and adjustment of the design 
document isa necessary prerequisite forlhe smooth running of the project. 
Thb moans that procedures for evaluation should be planned into the 
design document. At the same tinu?, the feedback from evaluation will 
mform improvements and adjustments to the document, both in rclaHon to 
the current course and with future courses in mind. Consequently there 
need to be channels which allow this information on the running of the 
programme to filter back to the designers. 

Although there are differences in emphasis between these three views 
on what should happen with evaluafion infonnation, they all consider that 
a project, course or curriculum needs to be well-informed abt>ut the 
realizaHon of the plan. This implies that it is necessary to create reliable and 
efficient channels for the dissemination of informaHon, wherever it is 
needed. 

i 



2.9 What kind ol constraints can we expect to encounter in the 
evaluation of a programme? 

It is generally rccc^zcd that the evaluation of a communicative 
curriculum is pr^Icmatic. The lack of an ^^ventoiy of items to be learned 
means that the assessment of a programme is less straightforward and 
tt^refore more difficult to operate. TMs helps to expl^n why there is a lack 
of well worked K)ut models of programme design for the omununicati ve 
curriculum that include evaluation^ especially of the formative kiml. 

In fact, judging from the literature available, thaie has up to now been 
relatively little interest in or investigation of the wk^le area of on-going 
evaluation. While there are many papers on ''evaluation", for tite majority 
of writers this means testing student performance in a more or less 
summalivc way* Indeedjt seems that what is called ''on-going, formative" 
evaluation is not always what it is meant to t>e. As Potts (1985, p25) points 
out, if on-going assessment does not influence subsequent activi^ and 
syllabus content, then it is in fact not a formative assessment, but a series of 
''mlcnvsummative" evaluations. 

Given that the planners of a project agree that an on going evaluation 
plan needs to be built in, there are still difficulties to be faced. Firstly, the 
capacity of those involved in the evaluation needs to be taken into account. 
A major problem here will he that teachers, learners, course designers, and 
administrators will be used to implement summative evaluation proce- 
dures and may not have the training to plan or carry out formative 
evaluation. A further constraint might be time — the time required for 
gathering and analysing data, and the time required lo use the information 
gathered in order to shape further development of the course in progress. 
Allied to this point is theamount of effort and resources that it is practicable 
to expend on this aspect of course planning. Another constraint concerns 
the channelling of information between participants in the evaluation. This 
may have many causes: 

- physical distance; 

- personality clashes and conflict within the hierarchy; 

- inadequate channels of aimmunication; 

- traditi()nal role relationships within the wider educational environ- 
ment. 

Any one of these, cir a combination of them, could make the kind of 
evaluation prcxredure we have been discussing here very difficult to 
implement. 

2.10 Conclusion 

The three writers on whom we have based our study in this section were 
chosen Invauso of their commitment to the implennentatiim of pnxredures 
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for formaHve evaluaHon. They have provided us with a thorough exposi- 

tfon of thereasonsforon-gofngevaluaHonandof issues thathavcto be taken 
into account if this kind of evaluation is to be applied to project design. We 
are now in • position to summariie the main features of a comprehensive 
system of evaluation which can play an effccHve role in programme 
planning and implementation: 
(i) The evaluation needs to be broad enough in its scope to be able to 
validate the programme design and the thcoreHcal concepts behind 
it, as well as to assess the success of the programme and of the 
learners. 

(li) It needs to provide a focus on process as well as on product, 
(iii) It needs to be ba^ed on clear criteria which all participants are aware 

of and which eva7uator5 can apply in analysis. 
Uv) It needs to involve all participants, non-expcrls as well as experts, in 

programnw design and modification, 
(v) It needs to specify methods of data collection wluch are appropriate 

to the context, to what is being evaluated, and to whoever is 

evaluating. These methods must be reliable and accessible to all 

concerned, 

(vi) It needs to provide suitableand reliable channels to f««d thcinforma- 
Hon collected back into the planning of the programme. 

(vii) It needs to include fon.^.iti ve as well as summaH ve procedures, to be 
planned into the programme from the start and to be carried out at 
regular intervals during and after the course. 

(viii) It needs to take into account all potential constraints that might arise 
in the context of a programme, and the design and evaluation of the 
programme should be adjusted accordingly. 

If we attempt to incorporate these features into a project, the task of 
providing for evaluation will become a much more rigorous and demand- 
ing one than we have hitherto known. We must therefore go on to ask 
ourselves if this effort would really pay dividends in the form of improve- 
ments to our projects. Do we really want to expend so much time, energy, 
and resources on developing alternative procedures for evaluation? Would 
the data we collect enable the programme to become more efficient or 
effective than present programmes, which depend on well-dcsicned tests 
and aware and interested teachers? 



3 EVALUATION AND INNOVATION 

So far in this paper we have Itxiked at projtnt design from various 
perspectives. In the first section we examined a variety of project types to 
sec what elements of evaluation they contained, and in the second section 




we looked at some theories of evaluation in project design. Now we 
consider another aspect of project design, namely the extent to whidi 
projects introduce innovation. We begin by examining different features of 
innovation in projects and discuss the extent to which planning of change 
is necessary in project design. Then we go on to examine the contribution 
that the type of evaluatimi we described at ^e end J the previous section 
could make to the planning of innovation. 

For ideas on innovation in ptt^ design we turn to Kennedy (1988). The 
starting point of his argument is dial all English language teaching projects 
imply change to a greater or lesser d^ee. Thus he argues that in planning 
and managing projects skills are needed for this aspect of programme 
de^gn. He stresses the need to manage change In the initial stages of a 
project, during project implementation, and on comp:etion (Kennedy 1988, 
pp329f.). 

We can already establish a link here with the type of evaluation tiiat is 
suggested by our discussion so far. Therefore a further reason why we 
should plan for on-going evaluation might be in order to manage change. 
However, if we are going toadopt Kennedy'sargument as a justifkation for 
on-going evaluation, we need first to clarify what we mean by innovation 
in English language teaching projects and then to plan how evaluative 
procedures might be ti^ed to manage change 

3.1 Innovation and the project 

a) Plans to introduce innovation 

A project is essentially a means by which we introduce innovation Into 
a system. If we look back over the projects which we examined in section 
1 , we see examples of desired cha ngcs, either in the form of the introduction 
of something entirely new (iheStudy Skills in English project and the Somali 
textbook project) or by modi fying something that already exit ts (the Sierra 
Leone project, the Ain Shams project). 

These projects and the change they brii.g with tl^m proceed in defined 
stages. Kennedy (1988, p 329) broadly categorizes these stages as: 

- identification of a problem; 

- selection of the innovation; 

- incorporation, acceptance and diffusion of innovation. 

An example of this can be seen in the Somali textbook project. The need 
to develop new materials was identified by the Somali ministry, the 
innovation was decided on and worked out by KELT officers, and the 
resulting new materials and changes in teaching methods were introduced 
gradually and systematically as part of the programme. Kennedy (1988, 
p.329) makes the point that "changes brought about will not only be 
quantitative, they will also be qualitative, they will involve people as well 
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as things". Therefore, in the case of the Somali project, it was not only the 
oiiganization of material resources that needed to be managed, but a great 
deal of sensitivity was necessaiy in the management of the changes that 
resulted. 

Thiskindof sensitivity was alsostressed by Bowers when describing the 

AinShams prefect He too made thepointthatsitycurriculumdevelopment 
project is an attempt to introduce change *Tn • common set of interlocking 
professicnal and administrative systems* (Bowew 1983, p.lOO). A change 
atany point in the system entails somechangeateverypcantBowers(19%3, 
pp.lOOff .) describes the set of systems which curriculum innovation affects 

anddefinestheissueswhichariseineachcase.Hecomesupwitha''$pider's 
web" of interlocking systems which include personnel, facilities, hinding 
and management. Kennedy (1988, p J31 ) adds sodal, political and cultural 
systems to this list. "Whenever the spader's web b touched, the whole 
trembles" (Bowers 1983, p.l04). Bowers points to the need for awarenessof 
the effect of this interdependence on the project as a whole. 

b) Planning for (m-going innovation 

Change is clearly a feature of the life of a project, and this must be taken 
account of. If we accept the views of Kennedy and Bowers, then reliance on 

ana(//iOf system thatdcpendson thesensitivity or intuitionsof planners and 
administrators is not sufficient. We need a project plan that: 

- "can be monitored and adjusted as the process of change is takiig 
place" (Kennedy 1988, p. 330); 

- can look back over a project and relate the process to the ou tcomc in 
order to investigate reasons for successes and failures; 

- can take this information forward to future project development. 
An example of such a project plan is the Tanzanian syllabus and 

materials design programme described in section 1 (Brumfit 1980). The 
course was structured in such a way that change was not only allowed for 
but also encouraged. The frameworic for the course was not a fixed scheme, 
but was based on a syllabus and a set of principles tltat acted as a working 
document. This meant diat there was capacity for change during the 
running of the course. In the long term this could mean the replacement of 
the original document, a restructuring of the syllabus, and major changes 
to the teacher-training programme. 

c) Sysff ms affecting and affected by inmvation 

Plr.sining a project n\eans planning for innovation. We ha ve already seen 
that there are certain features of innovation that we need to keep in mind. 
Kennedy (1988, pp. 330f.) highlights some of these features: 
(i) the reciprocal effect of innovation on systems and systems on 
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innovation; 

(ii) the importance of innovations being acceptable to local conditions 

and cultural norms; 
(iti) the need to involve and consult all participants in the innovation 

process; 

(i v) the need to show that the individuals affected by the innovation will 
receive t>enefits as a result 
Given that a project will affect existing systems, we must also expect that 
these systems will exercise some influence over die project and the innova-* 
tion. Kennedy (1988,p.332) offers a plan of a hierarchy of systems in terms 
of relative influence: 



cultural 

I 

political 

I 

administrative 

I 

educational 
I 

institutional 

I 

classroom innovation 



The point here is that problems will arise 

- if the project and innovation management are ignorant of the higher 
order factors; 

- if they do not work within these factors; 

- If they try to change them. 

This therefore returns us to the point that we made earlier we need to 
manage change in tte initial stages of the pro|ect. But now we go a step 
further and advocate project pre-planning, at which stag^ the innovations, 
their effects on systems, and the way they might in turn be affected by 
existing s^stems^ will need to be predictrd in advance, and the whole 
context evaluated, before change is ever made. This then needs to l>e 
controlled throughout the whole project 

d) Condilicms necessary for innovatioriB to succeed and be accepted 
ii) A variety of participants will be affected by the innovation intro- 
duced by a project. The important question is wMher there is 
communication between the participants, and if so, whether that 
communication is one-way or two-way. 
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(U) All those involved in the project need to agree that innovation is 
necessaiy . For example, the teachers need to feel that there is a close 
matdi between their working context and the innovation at three 
levels: 

- feasibility in temns of resources and organisation; 

- acceptability in termsof theteachingand learningstyleadvocated 
by the project, and their own philosophy; 

- relevance in tenns ctf the needs ol the learners. 

(iii) Innovation needs to be incremental rather than sudden, so that all 
aspects of the programme can be adopted. 

(iv) ''Ownership''isatermKennedyusestodescribe''thcdegrectowhich 
the participants feel that the innovation Ijelongs' to them" (Kennedy 
1988, p338). This, he claims, hasa great influenceover the likeUhood 
of innovation establishing itself. 

(v) It is important that teachers feel they have something to gain from 
innovatksn resulting from a project, and that these gainsoutweigh the 
losses, Kennedy (1988, ppJ39-340) proposes a series of possible 
gains: 

- increased job security; 

- improved relations with the administration; 

- improved service to learners; 

- improvements to teachers' knowledge and skills; 

- intellectual satisfaction; 

- economic and professional rewards. 
These need to be off-set against such losses as: 

- the time-consuming nature of innovation; 

- the pressure on teachers to learn new skills; 

- the physical and mental resources required. 

3.2 Evaluation as the manager of innovation 
a) Plans to introduce evaluation 

The decision to introduce innovation into a system necessarily involves 
introducing some form of evaluation. This is the only means of measuring 
the effects of innovation as it occurs, making judgements about its success 
or failure, assessing its rqjercussions on other systems, and conseqi/cntly 
adjusting or continuing with the innovation plan. Accordingly, on-going 
evaluation needs to be introduced into a project plan in order to manage 
change from the outset. 



b) Planning for on-going evaluation 

Just as ad hoc planning for innovation is not sufficient, st) we need a 
systematic approaci to planning evaluation procedure. Just as innovation 
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requires a way of relating the proc^ ol change to its outcmne, so we need 
tobeabletorelatetheevdluationoflhepitxess(formative)tothee^^ 
of the outcome (si^mmative). If we wish our innovation to l>e carried on in 
the system dien this relationship Is vital. 

By providing for on-going evaluation in a programme plan, we are 
declaring from tl^ outset a willingness to accept diange, and to set up the 
right framework and the right atmo^here for It to occur in- Eventually this 
may result in a programvneplan that is very different frcm the initial design 
document. 

c) Systems affecting and affeciai by evaluation 

A knowledge of the systems within which the project is being con- 
structed is clearly necessary. But it cannot be expected tihat the project 
designer will have direct access to all relevant information* This can be 
overcome by designing an evaluation system that allows input from all 
levels of ihe system. However, we cannot assume that getting relevant 
information at any particular stage of the programme, for example at the 
outset, will l>esufflcient This is because the repercussions of innovation will 
continue throughout the programme. The evaluation system has to cope 
with thisand providecondnuousfeedbackon the way in which systems are 
reacting. 

d) Conditions necessary for participants to co-operate in the evaluation pi.^xie- 
dure 

(i) Our evaluation needs to be able to tell us how all participants are 
reacting to the project. In the top-down hierarchy, for example, there 
may be commu nication downwards to the teachers, bu t there may l>e 
no existing channels to allow informatior to filter back up frcm the 
classroom. The evaluator needs to be aware of this and to provide 
suitable channels. That nruiy mean, for example, that it falls to the 
course designers themselves to distribute questionnaires and other 
means of gathering information to the teachers. 

(ii) If teachers have a hand in the decision-making process concerning 
innovations and are asked through the evaluation process for their 
opinions, then th^aremorelikely to welcome theinnovation. Itmay 
happen that teachers still do not welcon^ the innovation, for a 
number of reasons. One major reason might be that the innovation is 
at odds with their own philosophy of teaching. Through effective 
evaluation it mightbe possible to identify tfiisproblemand asa result 
make necessary adjustments, for example to the teacher-training 
programme. 

(ill) As we have seen from Bowers's "spider's web'\ a change can bring 
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about a number of associated changes. An evaluation would be able 
to assess this ripple effect at regular intervals. This might involve 
introducing the innovation in stages and evaluating itseffectsateach 
stage throughout the prolamine. 
(iv) The idea that teachers make gains or losses as a result of innovation 
is very relevant to the evaluation of a profect Qearly, the evaluation 
procedure must be able to assess what burden it puis on participants. 

This burden can be increased or reduced according to the gains that 
participants may make. For this reason evalualicm will have to find 

out if there are gains to be nuKte. If there are not, it may be that the 
de^gner can promote some. ¥or example, in the Somali project, the 
creation of teacher associations fostered solidarity among teactei^. 
On the other hand, it may be impossible, because of existing systems, 
to create gains. In that case, the designer cmild aim at a reduction of 
losses. In any case, a project designer should be able to evaluate in 
advance whatgains could beantidpated for participants, and set the 
targets accordingly. 

33 Conclusion 

To the question: Is it worth taking a new look at evaluation procedures? 
the answer must be "yes". If we agree that projects should be set up, 
curricula instituted, materials written, teacher-training courses planned, 
and language courses implemented, all with the aim of changing some- 
thing; and if we agree that it is desirable md useful to monitor and shape 
thischange as it occurs; then we must hav e a suitable way of evaluating the 
change. There is a great deal in conunon between managing innovation (in 
Kennedy's 1988 terms) and conducting evaluation as it is envisaged in this 
paper. 



4 THE MALAYSIAN STUDY SKILLS PROJECT: 
TEACHER EVALUATORS 

Up to now we have been mainly concerned with assessment from the 
perspective of the course designer, who in most cases is an export ouisider. 
However, we have also made the point that evaluation can be conducted 
from the perspectiveof any of the participants in a project. To continue our 
inquiry into project evaluation we k»k at a project in which I was involved. 
It does not seem sensible to examine it from the course designer's perspec- 
tive, since I could expect to have only a partial view of this- Instead I will 
examine the project from my perspective as a teacher implementing the 
project design in the classroom. 

There is a certain difficulty in writing about something that one has been 
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personally involved in. On the one hand, ti^re is a desire to avoid being 
critical of the wwk of fonner ocdleagues; on the <Mm hand, tiiere is the the 
tendency to look back widi fhe lieMfit of hindsight and suggest how tfiin^ 
should have been done. However, even with tfiese depots, it is perhaps 
useful to look at a project from tl^ point of view of participants at the 
chalkface and give a different perspective from th^ umbrella view of 
designers. 

c5ur purpose is to examine the framework of the project in order to 
identify and label the elements of evaluation that were built in or included 
later. To do this we will review the design document and the project in 
practice, in the light of evaluation features whidt we cim$io;?r essential to 
project implementation, and which we listed at theend of section 2. But first 
we offer some background information on the project. 

4*1 Study skills in English for A Level students in Malaysian residen- 
tial schools 

This project was concerned with Malaysian students who had com- 
pleted their secondary education and were now studying for British A 
Levels in preparation for entrance to British universiti^. Apart from their 
A Level studies, the students also followed a course in study skills in English 
(SSE) over the two years. This was a major component, taking up twelve 
hours per week and designed: 

- to facilitate A Level studies; 

- to prepare students for British university life; 

- to prepare students for a proficiency test in English required by 
British universities. 

The plan was to set up ten centres around Malaysia, in state residential 
secondary schools, and to admit approximately sixty Lower Sixth students 
to each centre each year. The Ministry of Education oversaw the project and 
funding came from the Malaysian Public Works Department (JPA), which 
in due course would finance students at university. 

A course was designed and implemented by a British organisation^ the 
Centre for British Teachers Ltd (CFBT), called in for this specific purpose. 
The CFBT provided a substantial number of the project personnel: 

- a project planner to produce the course syllabus and subsequently to 
act as professional ccnn'dinator; 

- a project director to perform administrative functions and liaise with 
relevant bodies; 

- regional project directors to supervise the smooth running of the 
operation in the ten centres around the country and monitor the 
implementation of the course in classrooms; 

- centre co-ordinators who were teachers, one per centre, working 



within the schools and Uaising between school administration, the 
CFBT head office, and the teachers; 

- teachers— all the SSE staff and approximately half the subject teach- 
ers on the A Level programme were CFBT expatriates. 
The administration in the individual schools retained a level of contn>I 
over the A Uvel programme. ITie head teacher at each centre was the 

immediatesuperior of aUteachersJncludingtheexpatriate staff. There was 
an A Uvel committee of all A Level teadiers^ expatriate and Malaysian. 
There wasan A UveloMjpdinator^a MalaysiancoJIeagueappointed by and 
answerable to thehead teacher. The AUvdcoordinator had responsibility 
for decisions made at each centre and for Uaising with the Mlnistiy and the 
JPA on behalf <rf his/her committee. 

4^ Assessing the evaluation policy of the project 

We shall look at each of the eight main features of evaluation that we 

listed at the end of section 2 and assess to what extent this project 

incorporated them. 

(0 The evaluation needs to be broad enough in its scope to be able to validate the 
programme design and the theoretical concepts behind it, as well as toassess 
the success of the programme and of the kartters. 

The design document allowed for assessment of student progress and 
course success. In Floyd (1986, p.60) task-based assessment is defined as 
follows: Trogress towards each course objective can be tested by setting 
specific pieces of work f...!. These tasks can be graded in difficulty and 
assigned a value on a band scale |...I. Alternatively, criteria can be estab- 
lished for performance at each band level on the same task." This plan for 
graded tasks aUows us to judge the progress of the course and the 
performanceofparticularstudentsintermsof product. If used in all centres, 
it would also allow evaluation of the relative progress in the different 
schools, which might be useful for external evaluation requirements. This 
Is probably the type of assessment that Potts (1985) would label "micro- 
summative" in that continuous assessment of student performance is 
provided for, but the focus of the evaluation is summative rather than 
formative. The criterion used for judging success is the extent to which 
students, and by extension the course, meet course objectives. 

Clearly, this document is based on theoretical concepts. While they are 

notexplicitly discussed in thedocumenl, they areevident in its contents. For 
example, it is taken for granted that participants will agree that such 
concepts as independent learning, task-based learning, integrated skills 
work, student self-evaluation, and banding form the basis for an efft-cUve 
appriMch. We have already argued that an evaluation prxxedure should be 
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sensitive to tiie possibility that the theoretical cxmcepts of the document 
might be in conflict with participants' phUosoi^ies. Therefore some means 
of evaluating this match or mismatch needs to be present so that in the case 
of conflict an adjustment can be made to die document or changes made to 
teacher recruitment/training. 

As far as valldatfon of the thinking behind the document is concerned, 
the document provides us witti an q;^x»rtunity to get information on 
whether the approaches to learning and teaching inheroit in the project are 
working (Floyd 1986. ppJSZ^). The most obvious way of getting such 
information is probably studait self-assessment with fe^back to SSE 
teachers and eventual improvement of die course as the goal 

In practice the project followed the design document ctosdy as far as the 
monitoring of student progress was concCTned. As regards evaluation of 
the theory behind the course, channels were planned in to gather teacher 
and student reaction throughout the running ^ the course. For example, at 
the beginning of the project and at each new intake of teachers, a two^weck 
orientation course provided an introduction to the thinking and method 
behind the course in workshop sessions^ Subsequently, there were a 
number of possible channels for teachers to feed back their enervations on 
approaches and methods being t^ed on the course: 

- regular seminars and workshopsbased on contributions from teach- 
ers and a ttended on a voluntary basis by teachers from all cen tres and 
by CFBT administrative staff; 

- regular visits to the centres by CFBT regional project directors 
(RPDsK- during these visits the opportunity was given for discus- 
sion of classrcx)m methods and materials, both on an individual and 
group basis and RPDs observed teachers in the classroom a nd wrote 
reports in the form of lesson descriptions; 

- a regular internal publication, A level Exchange, which contained 
^'^sson ideas, views on different approaches, and so on; 

-te*mly meetings between coordinators from all the centres and 
CFLT staff to discuss, among other things, the progress of the 
course — at these meetings co-ordinatore represented the viewsof the 
teachers at their centres, discussed and agreed subsequent changes 
to the programme, and carried news of these changes back to the 
centres; 

- meetings with teachers at the centres followed to discuss changes — 
reportsof these meetings were returned totheCFBT,wheredisagree- 
ments were noted; 

- from the point of view of one teacher at a particular centre, a very 
fruitful channel for the evaluation of appn^aches and the l)est source 
of feedback was the closely-kntt SSE department at that centre. 
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At a formal level the channel of communication was weekly meetings of 
the six-strong department This was a means of filtering out ideas to be 
communicated to the CFBT hierarchy through the avoidinators' meeting3 
and disseminated via the internal publication. However, it was really at 
classroom and staffroom level, through team lesson planning and team 
teaching, thatthe most useful evaluation of basicapproaches took place.Fbr 
variousreasons, diemorefbnnal channels listed above did not always work 
as they were intended ta For example, attendance at worksho^Ts was 
haphazard; and there is no doubt that distances between centivjj made 
communication difficult For the same reason, RPD visits we,« not as 
frequent as they might have been and often concentrated more on welfare 
concerns than dassroom matters. Also, a laige prt of the centre co- 
ordinator's time was spent on bureaucratic details which limited the time 
available for evaluating the progress of the course. 

(iO Evaluathn needs lo provide a focus on process as well as oti product. 
The design document accords great importance to the achievement of 
good A Level results and to the English proficiency exams. The target 
product of the course isa student who has acceptable levels in his academic 
subjects and inEnglish^and whoasaconsequeiKegetsa placeat university 
in Britain. However, to achieve this product, aHcntion to the learning 
process is vital and this is reflected in the SSE course objocHves: 
"1. To develop the language, thinking and independent learning skills 

needed to achieve the overall aims li e. success in A Levels and 

university entrance]. 
"2. To facilitate the transfer and application of these skills to A Level 

subject learning^ (Floyd 1986, p.l). 
The opportunity to evaluate the evolving learning process is built in 
through the self;assessment component in the form of self-assessment 
forms completed by shidents, learning diaries and tutorials. It is also 
allowed for in the continuous assessment plan through feedback from 
subject teachers and possibly through tasks to assess needs and progress, 
if these tasks are focussed on the process rather than on the product. 
However, the machinery for doing this was not built into the design 
document, and individual teachers were left to devise It. For instance, the 
document gives no indication of what a self-assessment form might look 
like; no criteria are given for evaluating information gathered from learners 
in diaries or on self-assessment forms; and no procedure is suggested for 
eliciting information from subject teachers helpful to the evaluation of 
process. This prompts us to make a general comment about the focus of the 
evaluation within the document There isa hea^ leremphasison monitoring 
and evaluating performance; and while some attenHon is paid to learning 
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procKses, there is far more focus on tht evaluatkm of product* 

In practice the project proved diat tiie design document assumed a lot 
on the part of the subject and SSU teachers involved. In fact, very little 
systema tic e valuaticm of the learning process took place at the outset of the 
course because of the lack of instniments and guidance However, as ti 
went on, within individual centres tte very diverse aims of the two 
directions of the course (i*e* A Level and SSE) began to mei^ge as teadiers 
worked together, discussed mutual problems, and sought solutions that 
were compatible with both setec^aima Thusitemei^ged tfiatjcrint attention 
totheprocessof l^mingand teaching was the most effectiveand consistent 
way of yielding an improved product i*e better English and sut^t resul ts. 

(in) tvaluatwn needs to be based on clear criteria which ail participants are 
aiijare of and which evaluators can opfAy in analym. 

The design document contains guidelines as to what is expected of 
student performance. These were expressed in band scales (Floyd 1986, 
pp.66-70), were widely circulated, and l>ecame familiar to students. 

We have already establis!^ thatthe design documentadvocatesa task- 
based approach. We would thereforeexpect to have at our disposal a range 
of criteria by which we could judge whether our students were able to 
perform the tasks, not just in terms of product, but also in terms of the 
process in vol ved in perfonning that task. However, the document does not 
provide this in a very detailed way (Floyd 1986, p.26)* The idea is that the 
teacher should be able to use these criteria as a cheddist to note lx>th 
successes and failures and feed that information into sut>sequent lesson 
planning. One comment we neec". to make here is while we have guidance 
forthecollectionoftheevaluation^dta,thereisnoguiddncecmhowtomake 
use of it. In practice it proved straightforward to rely on the design 
document for performance criteria* For tests, the band scales came to t>e 
replaced by closely related ones, namely those of the English Language 
Testing Service exam. 

As far as the introduction of new learning processes was ccmcemed, the 
teachers had to evolve their own criteria for evaluation* These were based 
on teachers' own experience of the learning context in Britain and else- 
where, as well as observations by teachei^ of existing learning processes 
being applied to A Level n>a»rials in Malaysian classrooms. Teachers 
quickly observed that there was a greatgulf between the laming processes 
required by A Level and university study and those that students were 
actually applying. The latter wereclearly insufficient The criteria therefore 
were based on the teachers' own judgement of what processes were 
necessary and they observed learners to assess such things as: 
- whether they adopted a problem-solving approach; 




- whether they tended to team by rote; 

- whether they were capable of tackling problems in groups; 

- whether they could split up tasks into relevant sub-tasks; 

- wheth^ they were able to transfer skills from SSE to A Level. 

<w> The ewluation needs to imolve aU fmHcipants, non experts asivellt^ 
experts, in programme design and mixlification. 

The design document acknowledges that teachers contributed to it by 
submitting schemes of work, suggesting teaching Weas, and offering 
examples of materials. It goes on to say: i.„l to a la^ extent the syllabus 
is a reflection, now in booklet form, of the teaching which is already going 
on in the various centres" (Floyd 1986) 

This is an example of a design document which appeared after the 
implementationof theprogranune.Thedevelopma\tand theimplementa- 
tion of the programme started around the same time. Bachman and Strick 
(1981, p51) comment that this represents a typical sequencing of a new 
programme in which there is little or no time for development before 
implementation. The advantage of this is that there Is time to involve 
participants in the development, so that some evaluation of the document 
by participants is possible. 

The completed document contains suggestions for a teaching pro- 
gramme, but imposes no rigid structure. Therefore, in so far as teachers 
were able to use it as a framework, they designed and modified their own 
courses according to the feedback they obtained in their own classrooms. 

The design document indicate that subject teachers should also be 
involved in SSE programme design in so far as they are able to evaluate the 
current needs of students (Ftoyd 1986, ppjf.). In practice this proved 
possible, but it tooka long time to devek)p suitable means by which toelidt 
from subject teachers what exactly student needs were, in terms that SSE 
teachers could translate into study skills. What was lacking was an evalu- 
ation instrument the language of whteh was accessible to both SSE and 
subject staff. This kind of oo-involvement of subject and SSE staff in 
identiiying leamer needs varied from centre to centre and depended to a 
large degree on the cooperation Qxat existed between teachers. 

(v) The ewluation needs to specify methods of data collection which are 
appropriate to the context, to vMt is being evaluated and who is evaluating. 
These methods must he reliable and accessible to all concerned. 
The design document refers to a number of different means of data 
collection under self-assessment, peer assessment, continuous assessment, 
commtm tests, and external exams (Floyd, pp.62f.). All of these arc con- 
cerned with the evaluation of students' performancv only. 

In practice, methods of data collection also included introspection on the 
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part of teachers and observation of teacher perfonrance. This developed 
over time, but not on any formal basis. The methods of data collection which 
were used mc^t often were: 

• infbnnal teacher oteervatkm of own lessons; 

- joint observation during team-teaching and suteequent discussion; 

- observation of the measure of success/failure of lessons planned by 
other teachers and subsequent discussion; 

- RPD observation of individual lessons and feedback to the teacher 
concerned in the formof a report; 

- discussion with students on a tutorial baste about learning progress; 

- elidtation of feedbadc from students on their language ivBeds, the 
appropriacy of the content of the course to their needs, and their 
estimation of their own progress; 

' standaidized tests— including English placement tests used for 
streaming, tests in English and A Level subjects conuncm to all A 
Level centres, mock O Level and ELTS tests of English, actual O level 
and ELTS tests. 

( w) Etfoluation needs to include formative as well as summative procedures, to 
be planned into the programme from the start and carried out at regular 
intervals during and after the course. 

The design document as we have already seen, details types of summa- 
tive evaluation of student performance and suggest that formative evalu- 
ation procedures would beagood idea, butitdoesnotdetailthem.Themain 
evaluation that the documentdwellson is the test performance of students 
and this is summative. 

In practice, formative evaluation played a vital role in the programme. 
As we have seen, individual teachers, or teams of teachers, at the different 
centres had the task of designing their own teaching programmes on the 
basis of the loose framework of the design document. They were therefore 
highly reliant on their own evaluation of classroom performance. The 
success of methods and techniques reinforced existing approaches and 
failure led to a search for alternatives and nuxlificaticHi of the teaching 
programme. So, in effect, the looseness of the teaching syllabus encouraged 
this type of formative evaluation. It should be noted that the prefect 
recruited only qualified and experienced teachers to work on the SSE 
programme. We might conclude that this is a prerequisite if this type of 
evaluation is to be realized in this type of franwwork. Furthermore, the 
support provided by the organization in the form of professional develop- 
ment could also be considered significant 

(vii) The evaluation needs to provide suitable and reliable channels to feed the 
infonnation collected back into the j^anning of the programme. 



The design document does not distinguish clearly between methods for 
collecting data and channels for feeding this information back into the 
programme. As we have seen, th»e are suggestions for the gathering of 

information,butthcre is nodctallingofmcansforanalysingand re-utilizing 
this. This may be because the methods that are suggested are mainly carried 
out by teachers who havea direct involvement in the implementation of the 
programme. Therefore it is perhaps not fdt necessary to spedfy channels. 
The channel that the programme relies heavily on is the three-way commu- 
nicaHon process between subject teachers, SSE teachers and students. 

In practice the problem was that good conununication tended to rely to 
somedcgreeon good personal relationsbetween teadiers, rather than being 
provided for in a formal way in a programme. This can be seen in Rovd 
(1986, p.48). ^ 

The last stage of the flowchart indicates that evaluation should take 
place, but precisely what should be done with the informaHon is not 
specified. Ways l^ which this channel actually operated included: 

- the subject teacher observing students performing a subject-related 
task in the SSE classroom to allow him/her to note recurring prob- 
lems with subject material and incorporate this into his/her pro- 
gramme; 

- the subject teacher observing sUidents successfully performing ac- 
tivities in the SSE classroom, using techniques that he/she could 
adopt for the subject classroom (e.g. a group approach to problem- 
solving which was largely developed in the SSE classroom and 
carried over to subject teaching/learning through this channel); 

- the SSE teacher observing students in the subject classroom and 
noting language and study skills problems which were subsequently 
drdit with in SSE lessons; 

- the marking by subject teachers of subject-related assignments set in 
SSE lessons and subsequent feedback into subject and SSE pro- 
granunes; 

- the provision for one-to-one or group dialogu*^ between students 
and teachers to discuss current pr<*lems and possible solutions. 

(viii) Evaluation meds to take into account all potential constraints tluil might 
arise in the context of a particular programtne, and adjust the design of the 
programme and its evaluation accordingly. 

I cannot comment on the constraints that shaped the design dtKument 
since I was not involved in that aspect of the programme. Instead, my 
comments will be confined totheconstraintsthat were observed in practice 
and how these shaped the evaluation procedures used. 

The first constraint that we experienced was a varlatiim among the 
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participants in their capadty to put this type of evaltiaticm into practice. 
Allied to this was a great diversity in teaching phitosq>hies, and without 
wishing to oversimplify, this generally divided SSB from subject tochers. 
In some cases this renrtained an obstacle throughout the project and meant 
that summative evaluation remained dominant in c^tain areas of the 
programme. In other cases there was a ooming togetiier of approaches 
whidi eventually meant that more formative evaluatkm was possiMe. This 
was facilitated by a professiorwJ devek^xnent programme which allowed 
teachers, both subject and SSE, the oppCMlunity to discuss their ideas and 
methods. Related to this was the need to involve students as participants in 
self-evaluation. For cultural and educational reasons this had to be a slow 
process. The solution was to take an incremental ap(m>ach and introduce 
the method gradually. 

Another constraint was time, in the sense that gathering infcmnation 
could be a lengthy process. Allied to this are the effort and resources 
involved in collecting thedata and re-planning the course accordingly. Our 
observation was that i f changes came down through the hierarchy, then the 
constraints of time and effort could be obstacles to the evaluation process. 
However, for decisions taken at centre level, the same machinery was not 
necessary, the constraints were not as serious, and the changes to the 
programme were relatively rapid and effective. 

The one disadvantage of decision-taking at a local level was perhaps a 
lack of systematicity- Furthermore, there was a tendency to ''re-invent the 
wheer at each of the centres, simply because results of evaluation were not 
filtering back to the hierarchy and being disseminated to the centres in a 
usable form. Very often, information that had travelled these channels was 
so idiosyncratic and personalized that it was comprehensible and relevant 
only to those from whom it originated. 

It could be argued that a major reason why time at a local level was not 
a seriousconstraint wasbecauseof what Kennedy (19S8, p338) calls the loss 
and gain calculation. There is no doubt that teachers involved in evaluation 
were expending time and enefgy over and above the demands of their 
contracts. The system relied to a certain extent on the personal satisfaction 
that teachers obtained from this type of involvement. But it also provided 
"gains" to off-set loss"' through opportunities for professional develop- 
ment and other incentives to cooperative working and seminar participa- 
tion. 

We have already mentioned that the loose framework of the design 
document relied to a great degree on co<>peration between teachers at the 
centres. Whileon theonehand the successful workingof these relationships 
could provide the "motor"' foreffective on-going evaluation, breakdown in 
the relationships could be detrimental to this aspect of the prograntme. This 
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meamawtsuchfactorsaspei™alily,experieiKeandteachjngphilo»^ 
played a very important role. *^ ' 

4J3 Conclusion 

We can see that the Malaysian Study Skills project, with its comprehen- 
sive design document, does contain a number of elements of the type of 
evaluation we would see as important At the same Hme there is a need to 
translate these ideas and plans into a format which is readily usable by the 
participants, written in a language that is common to all, and which 
provides simple methods of analysis to make the best possible use of the 
data. 




5 GUIDEUNES FOR A PRACTICAL APPROACH TO ON-COING 
EVALUATION 

E)rawing on our understanding of what on-going evaluation should 
Ideally involve, considering the theoretical implications of implementing 
such a programme, taking into account that a driving force behind evalu- 
ation should be the management of change, and finally havi.ig looked at an 
example of a working system of evaluation, we can now attempt to draw 

upasetofguidelinesforapplyingon-golngevaluation procedures. Ouraim 
here is to make suggestions that would bring the system of evaluation 
described in secHon 4 more into line with the ideas explored in the previous 
sections. At the same time we intend that the suggestions made should be 
appropriate to a broad range of slniaHons. An important consideraHon in 
formulating these guidelines is to put forward ideas which are based on the 
theory that we have discussed, but which are at the same time immediately 
applicable in real contexts. 

Guideline 1 

First and foremost, what we are concerned with here is the pnxiucHon 
of a design document which offer?' ^s much support as possible for methods 
and techniques of evaluation since: 

- the design document and syllabus is the one element of a project 
which has the potential to reach all participants; 

- Itcan transcend problems posed by poor channels of communication 
such as geographic distance between centres or personality clashes 
within a centre; 

- it is a starting point for all discussions about the programme; 

- it can be a prop when participants need ideas on methods and 
tt'clmiques; 

- it can be a launching pad for further developments; 
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- it is a source of continuity when peiwnnel at various levels chai^ge 
or ideologies are thrown into question. 

Fbr all these reasons the de^gn document is veiy important. Therefore 
every thing which is important to the project sh:>uld be induded, explain 
and /or e)(emplified in it. Since we feel that evaluaticm is one of the 
important elements in any project, it too needs to beexplained as clearly and 
fully as possible in thedesign document. Fbr this reason the suggestions that 
follow are all related to the design document. 

Guideline Z 

The design document should make clear what teaching philosophy lies 
behind the programme and what this might involve in teims of approaches 
to teaching and learning, and the selection and use of methodologies ^^nd 
materials. By extension, this will involve making explicit the approach to 
evaluation U^at the document takes. 

It is important for the evaluation of the programme that participants 
should be aware ci the thinking behind the design in order to be able to 
contribute to the evaluation of that design. If a teacher does not understand 
why a p^irticular type of activity is present in a programme, there is every 
possibility that he/she will fail use this activity correctly and conse- 
quently will not be able to fudge whether the activity was successful or not. 
Furthermore, the objectives of a course will be closely linked to the 
teaching/learning philosophy behind it. If the access of our course is 
measured according to the extent to which learners meet theobjecti ves, then 
we must rely heavily on all participants understanding what meeting the 
objectives involves. And to do that they need to understand the theoretical 
concepts behind them. 

To fulfil these requirements the design document should: 

- include an explanation of the theory, and of associated approaches 
and activities; 

- express that explanation in a language and format accessible to all 
participants, expert and non-expert alike; 

- make clear that the principles set out in the document are not fixed 
but are open to change; 

- list suggested techniques for evaluation. 

Guideline 3 

The document must provide participants with the language they need 
to collect, understand, and analyse data relevant toevaluation. Unless there 
is a common language theit? will always be the problem that data collected 
by different participants will vary and will only be useful to those who 
collected it. Consequently it will not be easy to analyse the available data 
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centrally and disseminate it for wider use. Means of providing a common 
language might be: 

- to prescribe precise methods of data collection; 

- to provide examples of methods of data collecHon which are repro- 
ducible and usable as they stand, e.g. questionnaires, checklists, 
assessment forms; 

- to recommend the submission of certain data to the managing body 
at regular intervals. 

Guideline 4 

The document needs to provide criteria for the evaluation of process 
which are just as explicit as those for the evaluation of product. These 
criteria are not only the domain of e-jperts (course designers, administra- 
tors, English or SSE teachers), but should be available to all parHcipanls, 
including students and subject teachers. TTie design document should give 
lists of criteria for the evaluation of process that can be reproduced for 
students and teachers. These can then serve as a basis for discussion in the 
light of the experience of participants during the programme. 

Guideline 5 

The design document needs to provide participants with dear ideas on 
how collected data might be analysed For this participants need to under- 
stand the valueof certain typesof data and how much informaHon thisdata 
can give about the progranune. In order to do this, the document could: 

- conUin worked examples of data and their analysis; 

- offer ideas on what particular results might signify for a programme; 
-suggest what remedial action ntight be taken based on particular 

results; 

-recommend how results can act as a basis for discussion of and 
change to a progranune; 

- show how student access to those results can act as a useful basis for 
individualized learning and self-assessment. 

Guideline 6 

Thedesign document needs to list, describe and explain the i mpHcations 
of all the possible channels that could be used to feed information back into 
the programme. This will mean: 

- giving precise instructions on how to use the various channels, for 
example prescribing how often teachers might meet; 

- making explicit theaimsof using certain channels, especially in cases 
where participants mighlbe mistrustful (forexample, teachers should 
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have access to examples of lesson observation sheets, which could be 
a useful channel for disseminating good lesson descriptions and 
ideas); 

- making explicit the overall plan the coordination of channels; 

- making explicit inc way that formal channels work— for example, 
explaining how Mi.tishfy decisions are incorporated into the pro- 
gramme, or how information from teachers is processed. 

Guidelhte 7 

The design docunr^nt needs to include a fiilly worked out and clearly 
explained process for thesystematicredevelopmentof theprogranwneand, 
by extension, of the design document itself, based on information from the 
evaluation system. This would invc^ve: 

- specifying a timetable for staged reassessment of the programme; 

- specifying a procedure for the collation of the data from the whole 
evaluation machine; 

- specifying how the consequent changes to the programme will be 
decided on. 

Guideline 8 

The design docunwnt must make clear that this approach to evaluation 
is an incremental one. What happens at the beginning of the programme 
might not necessarily be what happens at the end, and dianges will depend 
on the system of evaluation to which all participants may make a contribu- 
tion. In other words, although the document prescribes procedures, this 
does not mean that they will necessarily always remain in place. But at the 
same time the docunvent must make clear thai changes to these procedures 
should be discussed and justified. 

As a result of these suggestions, the design document would inevitably 
be a larger document than is usually the case; and Its role would be more 
substantial than that of the usual course syllabus. Indeed, the document we 
propose here might be described as a "course companion". Certainly such 
adesign document would become a vital tool in the planning, management, 
implementation and evaluation of the course which was based on it. 
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